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February  28,  2002 

The  Honorable  Joe  Knollenberg 
Chairman 

Subcommittee  on  the  District  of  Columbia 
Committee  on  Appropriations 
House  of  Representatives 

Dear  Mr.  Chairman: 

This  letter  responds  to  your  August  20,  2001,  request  that  we  perform  an 
initial  review  of  the  District  of  Columbia  Courts’  (DC  Courts)  effort  to 
acquire  a  new  information  system.  Faced  with  a  myriad  of  nonintegrated 
systems  that  do  not  provide  the  necessary  information  to  support  its 
overall  mission,  the  DC  Courts  is  in  the  process  of  acquiring  a  replacement 
system  called  the  Integrated  Justice  Information  System  (UIS).  This 
system  is  expected  to  address  the  current  system’s  deficiencies  and 
provide  DC  Courts  with  the  information  it  needs  to  perform  its  mission. 
The  District  of  Columbia  Appropriation  Act,  2001,^  provides  that  none  of 
the  funds  in  the  act  or  in  any  other  act  shall  be  available  for  the  purchases, 
installations,  or  operation  of  UIS  until  a  detailed  plan  and  design  has  been 
submitted  by  DC  Courts  and  approved  by  the  House  and  Senate 
committees  on  appropriations.  DC  Courts  is  in  the  initial  stages  of  the 
system  acquisition  effort — developing  a  request  for  proposal  (RFT)  to 
solicit  bids  from  vendors  for  the  design  and  implementation  of  the  new 
system.^  As  required,  this  detailed  plan  and  design,  which  includes  a  draft 
RFP,  was  submitted  to  the  appropriations  committees  on  May  17, 2001,  for 
review. 

Specifically,  we  assessed  whether 

1.  DC  Courts  has  implemented  the  disciplined  processes  for  this  project 
to  reduce  the  risk  associated  with  this  effort  to  acceptable  levels; 


^Public  Law  No.  106-522, 114  Stat  2440, 2442  (2000). 

^An  RFP  is  a  formal  request  for  vendors  to  provide  a  solution  to  a  stated  problem,  in  this 
case,  a  system  to  handle  DC  Courts’  management  information  needs. 
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Results  in  Brief 


2.  DC  Courts’  requirements  to  acquire  the  system,  as  identified  in  its  draft 
RFP,  contain  the  necessary  specificity  to  reduce  the  risks  from 
requirement  defects  to  acceptable  levels^  and 

3.  DC  Courts  has  performed  the  necessary  actions  to  determine  that  a 
commercial  off-the-shelf  system  (COTS)  would  meet  its  needs. 


DC  Courts  has  not  yet  implemented  the  disciplined  processes  necessary  to 
reduce  the  risks  associated  with  acquiring  and  managing  the  UIS 
acquisition  effort  to  acceptable  levels.  A  disciplined  software  development 
and  acquisition  process  maximizes  the  likelihood  of  achieving  the 
intended  results  (performance)  within  established  resources  (costs)  on 
schedule.  DC  Courts  officials  acknowledged  that  they  do  not  yet  have  the 
disciplined  processes  in  place  to  reduce  the  risks  from  this  effort  to 
acceptable  levels.  However,  they  also  acknowledged  that  disciplined 
processes  were  necessary.  They  further  noted  that  even  though  sufficient 
funding  to  fuUy  implement  disciplined  processes  would  not  be  available 
xmtil  the  system  was  approved  and  funded,  they  had  already  begun  to 
implement  some  elements  of  a  disciplined  process.  For  example,  at  the 
time  of  our  review,  DC  Courts  was  sending  several  people  to  be  trained 
and  certified  in  project  management  skills. 

The  m^gority  of  the  DC  Courts’  requirements,  developed  for  the  draft  RFP, 
lacked  the  necessary  specificity  to  ensure  that  the  defects  in  these 
requirements  have  been  reduced  to  acceptable  levels  and  that  the  system 
would  meet  its  users’  needs.  In  addition,  the  requirements  in  the  draft  RFP 
did  not  directly  relate  to  industry  standards  and  the  terms  “customization” 
and  “modification”  were  not  clearly  defined  in  the  draft  RFP.  We  also 
noted  that  the  system  requirements  were  not  logically  grouped. 

DC  Courts  officials  are  electing  to  use  the  acquisition  process  to  identify 
the  cost,  schedule,  and  performance  gaps  associated  with  their  effort.  DC 
Courts  officials  acknowledged  that  this  approach  generally  increases  risk; 
however,  they  concluded  that  the  benefit  to  be  obtained — accelerating  the 
implementation  of  a  badly  needed  system — justifies  those  risks.  At  this 
point  in  the  process,  we  concur  with  DC  Courts’  decision  to  use  the 


^Although  all  projects  of  this  size  can  be  expected  to  have  some  requirements-related 
defects,  the  goal  is  to  reduce  the  number  of  such  defects  so  that  they  do  not  significantly 
affect  cost,  schedule,  or  performance. 
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acquisition  process  to  identify  the  cost,  schedule,  and  performance  gaps  in 
this  case  because  of  the  following  mitigating  factors. 

•  DC  Courts  officials  concluded,  based  on  a  qualitative  analysis,  that  they  do 
not  have  unique  requirements  that  would  prevent  the  utilization  of  a  COTS 
product.  Furthermore,  they  recognized  that  a  gap  analysis  must  be 
performed  as  part  of  the  vendor  selection  process  to  identify  the  cost, 
schedule,  and  performance  impacts  of  each  vendor’s  product  before 
deciding  which  product  to  acquire. 

•  DC  Courts  is  still  in  the  early  stages  of  the  system  acquisition  effort  and 
has  not  yet  reached  the  critical  point  of  contract  award.  Therefore,  a  gap 
analysis  can  still  be  performed  prior  to  the  system  acquisition  to  ensure 
that  a  COTS  product  can  cost  effectively  meet  DC  Courts’  needs. 

•  The  requirements  that  had  been  developed  lacked  the  necessary 
specificity  to  perform  a  meaningful  gap  analysis.  Therefore,  the  time  spent 
on  attempting  to  perform  the  gap  analysis  before  issuing  the  RFP  may  not 
have  been  effective. 

As  with  any  effort,  alternative  approaches  need  to  be  analyzed.  In  this 
case,  DC  Courts  officials  believe  that  the  benefits  associated  with  the 
reduced  risks  of  performing  a  detailed  gap  analysis  before  the  RFP  is 
issued — having  a  greater  assurance  that  a  COTS  product  would  meet  their 
needs — ^were  outweighed  by  the  costs  associated  with  delaying  this  effort. 

During  the  coxuse  of  our  work,  DC  Courts  officials  stated  their 
commitment  to  go  forward  with  this  project  only  after  the  necessary 
actions  have  been  taken  to  reduce  the  risks  to  acceptable  levels.  We  are 
making  recommendations  to  help  ensure  that  DC  Courts  adequately 
(1)  adopts  and  implements  disciplined  processes  to  help  ensure  that  its 
systems  development  effort  is  successful,  (2)  defines  the  requirements 
necessary  to  develop  its  new  system  in  its  RFP,  and  (3)  improves  its  ability 
to  assess  potential  solutions.  In  commenting  on  a  draft  of  our  report,  DC 
Courts  agreed  with  our  findings  and  recommendations  and  said  that  it  has 
begun  to  implement  our  recommendations  to  ensure  the  successful 
acquisition  of  UIS  as  outlined  in  our  report. 


Background 


The  District  of  Columbia  Court  Reform  and  Criminal  Procedure  Act  of 
IGTC  (Court  Reform  Act)  transferred  jurisdiction  over  aH  local  judicial 


^Public  Law  No.  91^358,  84  Stat.  473  (1970). 
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matters  to  a  unified  court  system  for  the  District.  This  entity,  known  as  DC 
Courts,  includes 

.  the  Superior  Court,  which  is  the  trial  court  with  general  jurisdiction  over 
virtually  all  local  legal  matters,  including  criminal,  civil,  juvenile,  domestic 
relations,  probate,  and  small  claims  cases,  and 

•  the  Court  of  Appeals,  the  highest  court  of  the  District  of  Columbia,  which 
reviews  all  appeals  from  the  Superior  Court,  as  weU  as  decisions  and 
orders  of  D.C.  government  administrative  agencies. 

The  Court  Reform  Act  provided  for  the  creation  of  a  policy-making  body 
for  DC  Courts,  the  Joint  Committee  on  Judicial  Administration.  The  joint 
committee,  composed  of  the  two  chief  judges  and  three  associate  judges, 
submits  DC  Courts’  annual  budget  requests  and  is  responsible  for  DC 
Courts’  general  personnel  policies,  accounting  and  auditing,  procurement 
and  disbursement,  development  and  coordination  of  statistical  and 
management  information  systems  and  reports,  and  other  related 
administrative  matters.  The  joint  committee  appoints  the  executive  officer, 
who  manages  the  day-to-day  administrative  and  financial  management  of 
the  court  system  on  the  committee’s  behalf. 

The  National  Capital  Revitalization  and  Self-Government  Improvement  Act 
of  1997"  (Revitalization  Act)  changed  DC  Courts’  funding  process, 
noryudicial  employee  compensation,  and  functional  responsibilities.  The 
Revitalization  Act  provides  for  direct  federal  funding  of  DC  Courts.  The 
joint  committee  submits  DC  Courts’  budget  request  to  the  Congress 
through  the  director  of  the  Office  of  Management  and  Budget  (0MB).  In 
addition,  some  DC  Courts  activities  were  transferred  to  the  federal 
government. 


DC  Courts’  Effort  to 
Acquire  New  Information 
System 


In  October  1998,  the  Superior  Court  of  the  District  of  Columbia  launched 
the  UIS  project  to  upgrade  and  enhance  its  information  management 
capabilities  and  establish  a  unified,  fuUy  integrated  computer  system  that 
would  support  data  collection  and  exchange  for  all  types  of  cases 
processed  within  the  Superior  Court.  UIS  is  a  multiyear  initiative  with  a 
two-fold  purpose:  first,  to  improve  data  collection  and  exchange  within 
and  across  DC  Courts’  multiple  divisions,  which  process  different  types  of 
cases  and  provide  essential  support  services,  such  as  research  and 
development  and  information  technology,  and  second,  to  improve 


"PubUc  Law  No.  105-33,  Title  XI,  111  Stat.  251, 712  (1997). 
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interagency  data  collection  and  exchange  among  the  District’s  criminal 
justice  agencies.  Currently,  DC  Courts  information  management  resources 
are  divided  among  18  different  computer  systems  that  have  evolved  over 
the  past  20  years  in  response  to  the  differing  needs  of  particular  court 
divisions.  DC  Courts’  UIS  initiative  is  part  of  a  District-wide  effort  to 
improve  the  data  collections  systems  and  infrastructure  of  District 
criminal  justice  agencies  and  enhance  data  exchange  among  those 
agencies. 

The  District  of  Columbia  Appropriations  Act,  2000,®  provided  that,  of  the 
amounts  available  for  operations  of  DC  Courts,  an  amount  not  to  exceed 
$2.5  million  would  be  used  for  the  design  of  UIS.  Due  to  the  time  needed 
to  prepare  a  detailed  requirement  analysis  to  guide  the  system  design,  no 
contract  was  awarded  or  funds  spent  during  fiscal  year  2000.  As  a  result, 
DC  Courts  officials  requested  the  $2.5  million  to  design  the  new  system  in 
the  fiscal  year  2001  capital  budget. 

DC  Courts  has  financed  an  UIS  requirements  analysis  through  a  $350,000 
grant  from  the  U.S.  Department  of  Justice’s  Edward  Byrne  Memorial  State 
and  Local  Law  Enforcement  Assistance  Formula  Grant  Program  through  a 
subgrant  of  the  D.C.  Office  of  Justice  Grants  Administration.  DC  Courts 
subsequently  awarded  a  contract  to  conduct  the  UIS  requirements 
analysis,  with  the  following  objectives:  (1)  assess  the  DC  Courts’  existing 
technology  infrastructure  and  systems  and  core  business  processes 
supported  by  these  systems,  (2)  determine  critical  information 
management  needed,  and  (3)  recommend  technical  and  business  process 
solutions  that  would  cost  effectively  meet  these  needs.  In  September  2000, 
the  contractor  issued  the  final  report  on  the  requirements  analysis,  which 
was  conducted  from  January  through  August  2000.  The  UIS  plan  and 
design  prepared  by  DC  Courts  is  based  on  the  requirements  analysis 
conducted  by  the  contractor. 

DC  Courts  provided  its  detailed  plan  and  design  for  the  UIS  to  the 
chairmen  of  the  Senate  and  House  appropriations  committees  on  May  17, 
2001.  On  August  20,  2001,  we  were  asked  to  review  the  DC  Courts’  draft 
RFP  for  the  design  and  implementation  of  the  new  system,  in  conjunction 
with  the  subcommittee’s  review  of  the  plan  and  design,  before  DC  Courts 
began  the  formal  solicitation  process. 


®Pubnc  Law  No.  106-113, 113  Stat  1501, 1502  (1999). 
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Scope  and 
Methodology 


To  carry  out  our  objectives,  we  reviewed  and  analyzed 

DC  Courts’  plan  and  design  for  the  UIS,  including  the  draft  RFP  for  the 
design  and  implementation  of  the  new  system; 

DC  Courts’  annual  report; 

industry  automation  standards  published  by  the  National  Consortium  for 
Court  Junctional  Standards;^  and 

reports  produced  by  the  contractor  related  to  (1)  DC  Courts’  Integrated 
Justice  Information  System  Requirements  Analysis,  (2)  Business  Process 
Descriptions,  Data  Flow  Diagrams  and  Entity  Relationship  Diagrams, 

(3)  Inventory  of  Data  Elements,  and  (4)  Executive  Summary  (September 

2000). 

We  discussed  the  UIS  project  with  the  following: 

the  chief  judge,  Superior  Court  of  the  District  of  Columbia; 
a  member  of  the  Technology  and  Automation  Committee; 
the  executive  director,  DC  Courts; 
the  cleric  of  the  court; 

the  director  of  Information  Technology,  Information  Technology  (IT) 
Division,  Court  System; 

the  information  system  administrator,  IT  Division,  Court  System; 
other  DC  Courts  personnel;  and 

representatives  from  the  contractor  who  prepared  the  requirements. 

We  reviewed  and  analyzed  the  draft  detailed  requirements  prepared  by  the 
contractor  and  DC  Courts  personnel  and  compared  them  to  industry 
standards  on  selected  court  activities,  such  as  domestic  relations  and  civil 
cases.  We  also  reviewed  the  Clinger-Cohen  Act  of  1996;®  federal  policy 
governing  acquisition  efforts,  including  OJVIB  guidance;  and  guidance  and 


^The  consortium  is  a  subgroup  of  the  Conference  of  State  Court  Administrators  and  the 
National  Association  for  Court  Management  Joint  Technolo^  Committee.  Its  goal  is  to 
develop  functional  standards  for  case  management  information  systems  for  civil,  domestic 
relations,  criminal,  juvenile,  probate,  and  traffic  cases. 

^Public  Law  104-106.  The  Clinger-Cohen  Act  requires  agencies  to  analyze  their  missions 
and,  based  on  the  analysis,  revise  mission-related  and  administrative  processes,  as 
appropriate,  before  making  significant  investments  in  information  technology  used  to 
support  those  missions. 
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best  practice  literature®  that  the  Software  Engineering  Institute  (SEI)/®  the 
Project  Management  Institute  (PMI)/^  and  the  Institute  of  Electrical  and 
Electronics  Engineers  (lEEE)^^  have  issued  on  evaluating  information 
technology  investment. 

We  did  not  independently  verify  or  audit  the  cost  data  we  obtained  from 
DC  Courts  officials. 

Our  work  was  conducted  from  October  2001  through  November  2001  in 
accordance  with  U.S.  generally  accepted  government  auditing  standards. 
We  requested  comments  on  a  draft  of  this  report  from  the  Joint  Committee 
on  Judicial  Administration  in  the  District  of  Columbia  The  joint 
committee  provided  us  with  written  comments  that  are  discussed  in  the 
‘'DC  Courts  Comments”  section  and  are  reprinted  in  appendix  L 


®U.S.  General  Accounting  Office,  Assessing  Risks  and  Returns:  A  Guide  for  Evaluating 
Federal  Agencies'  IT  Investment  Decision-Making y  GAO/AIMD-10.1.13  (Washington,  D.C., 
1997);  U.S.  General  Accounting  Office,  Executive  Guide:  Creating  Value  Through  World- 
class  Financial  Management^  GAO/AIMD'00-134  (Washington,  D.C.:  2000);  and  U.S. 
General  Accounting  Office,  Information  Technology:  An  Audit  Guide  for  Assessing 
Acquisition  Risks,  GAO/IMTEC-8.1.4  (Washington,  D.C.:  1992). 

^®SEI  is  recognized  for  its  experience  in  software  development  and  acquisition  processes.  It 
has  also  developed  methods  and  models  that  can  be  used  to  define  disciplined  processes 
and  determine  whether  an  organization  has  implemented  them.  SEFs  stated  mission  is  to 
provide  leadership  in  advancing  the  state  of  the  practice  of  software  engineering  and  to 
improve  the  quality  of  ^sterns  that  depend  on  software. 

^^PMI  provides  global  leadership  in  the  development  of  standards  for  the  practice  of  the 
project  management  profession  throughout  the  world.  PMI’s  standards  document,  A  Guide 
to  the  Project  Management  Body  of  Knowledge  (PMBOK®  Guide),  is  a  globally  recognized 
standard  for  managing  projects  in  today’s  marketplace.  The  PMBOK®  Guide  is  approved 
as  an  American  National  Standard  by  the  American  National  Standards  Institute. 

^^Institute  of  Electrical  and  Electronics  Engineers,  Transactions  on  Software  Engineering 
(IEEE  Transactions  on  Software  Engineering,  volume  14,  number  10  1988). 
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DC  Courts  Has  Not 
Implemented  the 
Disciplined  Processes 
Necessary  to  Reduce 
Risks  to  Acceptable 
Levels 


DC  Courts  has  not  implemented  the  disciplined  processes  necessary  to 
reduce  risks  associated  with  managing  its  system  acquisition  to  acceptable 
levels.  However,  DC  Courts  recognizes  the  importance  of  these  processes 
and  plans  to  implement  them  once  funding  for  the  project  is  available. 

Disciplined  processes  have  been  shown  to  reduce  the  risks  associated 
with  software  development  and  acquisition  efforts  to  acceptable  levels  and 
are  fundamental  to  successful  systems  acquisition.  A  disciplined  software 
development  and  acquisition  process  is  needed  to  maximize  the  likelihood 
of  achieving  the  intended  results  (performance)  within  established 
resources  (costs)  on  schedule.  Although  a  “standard  cookbook”  of 
practices  that  will  guarantee  success  does  not  exist,  several  organizations, 
such  as  SEI,  PMI,  and  IEEE,  and  individual  experts  have  identified  and 
developed  the  types  of  policies,  procedures,  and  practices  that  have  been 
demonstrated  to  reduce  development  time  and  enhance  effectiveness. 


SEI  and  others  have  documented  the  positive  effects  of  disciplined 
processes  and  have  developed  methodologies  that  can  be  used  to 
determine  whether  an  organization  has  such  processes.  For  example,  SEI 
first  developed  the  Capability  Maturity  Model  (CMM)  to  determine  an 
organization’s  ability  to  develop  software  and  has  also  developed  a  CMM 
to  assess  an  organization’s  ability  to  acquire  software.  These 
methodologies  are  designed  to  determine  whether  an  organization  has 
implemented  the  types  of  disciplined  processes  that  can  lead  to  lower 
defect  rates  and  help  avoid  the  adverse  impacts  associated  with  common 
mistakes.  Organizations  that  have  focused  on  improving  their  processes 
have  found,  over  time,  that  they  are  able  to  reduce  their  time  to  market  by 
about  one-half  and  reduce  the  costs  from  defects  by  factors  of  3  to  10.^^ 


The  key  to  having  a  disciplined  system  development  effort  is  to  have 
disciplined  processes  in  multiple  areas.  For  projects  such  as  the  one  being 
undertaken  by  the  DC  Courts,  these  include,  at  this  stage  in  the  acquisition 
process,  project  planning,  requirements  management,  project 
management,  configuration  management,  risk  management,  and  testing. 
Effective  processes  should  be  implemented  in  each  of  these  areas 
throughout  the  project  life  cycle  since  constant  changes  occur.  Table  1 


^^Steve  McConnell,  Rapid  Development:  Taming  Wild  Software  Schedules  (Redmond, 
Wash,:  Microsoft  I^ess,  1996). 
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provides  a  brief  description  of  some  of  the  areas  that  appear  critical  to  the 
DC  Courts’  effort.*^ 

Table  1 :  Examples  of  Disciplined  Processes 

Discipline 

Description 

Project  planning 

Project  planning  is  the  process  used  to  establish  reasonable  plans  for  performing  and  managing  the  software 
project.  This  Includes  (1)  developing  estimates  of  the  resources  needed  for  the  work  to  be  performed, 

(2)  establishing  the  necessary  commitments,  and  (3)  defining  the  plan  necessary  to  perform  the  work.  Effective 
planning  is  needed  to  identify  and  resolve  problems  as  soon  as  possible  when  It  Is  the  cheapest  to  fix  them. 
According  to  one  author,®  the  average  system  design  and  Implementation  project  includes  about  80  percent  of 

Its  time  as  unplanned  rework — ^fixing  mistakes  that  were  made  earlier  in  the  project. 

Risk  management 

Risk  management  is  a  set  of  continual  activities  for  identifying,  analyzing,  planning,  tracking,  and  controlling 
risks.  Risk  management  starts  with  Identifying  the  risks  before  they  become  problems.  If  this  step  is  not 
performed  well,  the  entire  risk  management  process  may  become  a  useless  exercise  since  a  process  cannot  be 
managed  without  information  about  that  process.  As  with  the  other  disciplined  processes,  risk  management  is 
designed  to  eliminate  the  effects  of  undesirable  events  at  the  earliest  possible  stage  to  avoid  the  costly 
consequences  of  rework. 

After  the  risks  are  identified,  they  need  to  be  analyzed  so  that  they  can  be  better  understood  and  decisions  can 
be  made  about  what  actions,  if  any,  will  be  taken  to  address  the  risks.  Basically,  this  step  includes  such 
activities  as  evaluating  the  impact  on  the  project  if  a  risk  does  occur,  determining  the  probability  of  the  event 
occurring,  grouping  like  risks  together,  and  prioritizing  the  risk  against  the  other  risks.  Once  the  risks  are 
analyzed,  a  risk  management  plan  is  developed  that  outlines  the  information  known  about  the  risks  and  the 
actions.  If  any,  that  will  be  taken  to  mitigate  those  risks.  Risk  management  is  a  continual  process  because  the 
risks  and  actions  planned  to  address  those  risks  need  to  be  monitored  to  ensure  that  the  risks  are  being 
properly  controlled  and  that  new  risks  are  identified  as  early  as  possible.  If  the  actions  envisioned  in  the  plan  are 
not  adequate,  then  additional  controls  are  needed  to  correct  the  deficiencies  identified. 

Testing 

Testing  is  the  process  of  executing  a  program  with  the  intent  of  finding  errors.*"  Disciplined  system  development 
activities  recognize  that  testing  will  not  find  all  defects  even  though  well-designed  and  executed  testing 
programs  are  used.  For  example,  testing  performed  through  the  system  testing  phase  often  catches  less  than 

60  percent  of  a  program’s  defects.®  Consequently,  testing  alone  cannot  be  relied  on  to  identify  all  defects. 

“Steve  McConnell,  Software  Project  Survival  Guide  {Re6mor\6,  Wash.:  Microsoft  Press,  1998). 


“’Glendford  J.  Myers,  The  Art  of  Software  Testing  (New  York,  N.Y.:  John  Wiley  and  Sons,  Inc.,  1979). 
‘'See  footnote  13, 


DC  Courts  officials  agreed  that  they  had  not  yet  fully  implemented  the 
disciplined  processes  necessary  to  reduce  the  risks  associated  with  this 
project  to  acceptable  levels.  They  said  that  they  recognized  that  the 
implementation  of  the  disciplined  processes  was  needed  not  only  for  this 
project  but  for  other  projects  as  well,  and  that  effective  implementation 
would  take  time,  management  commitment,  and  funding.  They  also  noted 
that  they  had  begun  the  process  of  identifying  the  needed  funding  and 


list  of  other  processes  necessary  to  acquire  or  develop  systems  can  be  obtained  from 
SEI  at  www.sei.cmu.edu. 
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staffing  levels  and  that  they  were  sending  several  of  the  project  managers 
to  training  so  that  they  could  become  certified  project  managers,  which 
win  facilitate  their  knowledge  of  disciplined  processes. 


DC  Courts  System 
Requirements  as 
Specified  in  RFP  Are 
Inadequate 


Requirements  represent  the  blueprint  that  system  developers  and  program 
managers  use  to  design,  develop,  and  acquire  a  system.  Requirements 
should  be  consistent  with  one  another,  verifiable,  and  directly  traceable  to 
higher-level  business  or  functional  requirements.  It  is  critical  that 
requirements  be  carefully  defined  and  that  they  flow  directly  from  the 
organization’s  concept  of  operations  (how  the  organization’s  day-to-day 
operations  are  or  will  be  carried  out  to  meet  mission  needs).  Improperly 
defined  or  incomplete  requirements  have  been  commonly  identified  as  a 
root  cause  of  system  failure  and  systems  that  do  not  meet  their  costs, 
schedules,  or  performance  goals.  Without  adequately  defined  requirements 
that  have  been  properly  reviewed  and  validated,  significant  risk  exists  that 
the  system  will  need  extensive  and  costly  changes  before  it  will  meet  the 
DC  Courts’  needs. 


DC  Court  system  requirements  set  forth  in  the  draft  RFP  lacked  the 
necessary  specificity  to  ensure  that  the  defects  in  these  requirements  have 
been  reduced  to  acceptable  levels  and  that  the  system  would  meet  its 
users’  needs.  In  addition,  the  terms  “customization”  and  “modification,” 
though  used  in  the  RFP,  were  not  clearly  defined.  Also,  industry  standards 
that  appeared  to  be  related  to  the  DC  Courts  were  either  not  used  or  were 
summarized  so  that  the  detail  provided  in  the  standard  was  omitted  from 
the  DC  Courts’  requirements.  We  also  noted  that  the  requirements  were 
not  logically  organized — ^related  requirements  were  not  grouped  together. 
Better  grouping  of  requirements  would  help  the  DC  Courts  and  potential 
contractors  in  assessing  whether  the  RFP’s  requirements  adequately 
describe  the  functionality  necessary  to  conduct  court  business.  Because 
DC  Courts’  requirements  were  not  specific,  were  poorly  organized,  and  did 
not  directly  relate  to  industry  standards,  a  significant  potential  exists  that 
the  DC  Courts’  proposed  system  may  not  meet  its  business  needs  or  may 
not  be  delivered  on  schedule  and  within  budget  if  corrective  action  is  not 
taken. 
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Requirements 
Management:  A  Key 
Process  for  Quantifying  a 
System’s  Purpose  and 
Faction 

Good  requirements  have  several  common  characteristics:^® 

•  They  fiilly  describe  the  software  functionality  to  be  delivered. 
Functionality  is  a  defined  objective  or  characteristic  action  of  a  system  or 
component.  For  example,  a  system  may  have  inventory  control  as  its 
primary  functionality.^^ 

•  They  provide  the  source  of  the  requirement.  For  instance,  the  citation  of 
the  statute,  regulation,  or  industry  standard  should  be  shown  so  that 
others  can  evaluate  the  applicability  of  the  requirement  and  better 
imderstand  the  impacts  of  changes  in  the  requirement. 

•  They  state  the  requirement  in  clear  terms  that  allow  for  quantitative 
evaluation.  Specifically,  all  readers  of  a  requirement  should  arrive  at  a 
sin^e,  consistent  interpretation  of  it. 

Studies  have  shown  that  problems  associated  with  requirements  definition 
are  key  factors  in  software  projects  that  do  not  meet  their  cost,  schedule, 
and  performance  goals.  For  example,  see  the  following: 

•  A  study  found  that  getting  a  requirement  right  in  the  first  place  costs  50  to 
200  times  less  than  waiting  until  after  the  system  is  implemented  to  get  it 
right.^® 


Requirements  management  is  a  process  that  establishes  a  common 
understanding  between  the  customer  and  the  software  project  manager 
regarding  the  customer’s  business  needs  that  will  be  addressed  by  a 
project.^®  A  critical  part  of  this  process  is  to  ensure  that  the  requirements 
development  portion  of  the  effort  documents,  at  a  sufficient  level  of  detail, 
the  problems  that  need  to  be  solved  and  the  objectives  that  need  to  be 
achieved. 


^®Camegie  Mellon  University  Software  Engineering  Institute,  The  Capability  Maturity 
Model:  Guidelines  for  Improving  the  Software  Process  (Reading,  Mass.:  Addison  Wesley 
Longman,  Inc.,  1995). 

^®Karl  E.  Wiegers,  Software  Requirement:  Practical  techniques  for  gathering  and 
managing  requirements  throughout  the  product  development  cycle  (Redmond,  Wash.: 
Microsoft  Press,  1999). 

^TEEE  610.12-1990. 

^®Barry  W.  Boehm  and  PhOip  N.  Papaccio,  Understanding  and  Controlling  Software  Costs 
(IEEE  Transactions  on  Software  Engineering,  volume  14,  number  10  1988). 
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•  A  survey  of  more  than  8,000  software  projects  found  that  the  top  three 
reasons  that  projects  were  delivered  late,  over  budget,  and  with  less 
functionality  than  desired  all  had  to  do  with  requirements  management^® 

•  A  study  found  that  the  average  project  experiences  about  a  2&-percent 
increase  in  requirements  over  its  lifetime,  which  translates  into  at  least  a 
25-percent  increase  in  the  schedule.^® 

•  A  study  noted  that  between  40  and  60  percent  of  all  defects  found  in  a 
software  project  could  be  traced  back  to  errors  made  during  the 
requirements  development  stage.^^ 


Requirements  Were 
Not  Specific 


The  majority  of  the  requirements  in  the  draft  RFP  lacked  the  necessary 
specificity  to  ensure  that  the  defects  in  the  requirements  had  been  reduced 
to  acceptable  levels.  Acceptable  levels  refer  to  the  fact  that  any  systems 
acquisition  effort,  such  as  that  being  undertaken  by  the  DC  Courts,  will 
have  some  requirements-related  defects.  However,  the  goal  is  to  reduce 
the  risks  and  prevent  significant  requirements  defects  in  order  to  limit  the 
negative  impact  of  these  defects  on  the  cost,  timeliness,  and  performance 
of  the  project.  The  lack  of  specificity  of  the  requirements  contained  in  the 
draft  RFP  indicate  that  a  significant  number  of  requirements-related 
defects  are  present  in  this  document.  These  requirements-related  defects 
significantly  increase  the  likelihood  that  the  resulting  system  will  not  meet 
DC  Courts’  cost,  schedule,  and  performance  goals.  In  addition,  the  risk 
exists  that  the  DC  Courts  may  have  failed  to  include  critical  requirements 
because  of  the  lack  of  specificity.  Omission  of  critical  requirements  wiU 
also  adversely  affect  cost,  schedule,  or  performance  goals.  As  noted 
elsewhere,  DC  Courts  officials  have  agreed  with  our  assessment  and  have 
started  tal^g  actions  to  address  these  weaknesses. 

DC  Courts’  requirements  lacked  the  specific  information  necessary  to 
understand  the  required  functionality  that  a  vendor  should  provide  and 
how  to  determine  quantitatively,  through  testing  or  other  analysis,  whether 
a  vendor’s  product  would  adequately  address  the  DC  Courts’  needs.  For 
examples,  see  the  following. 


^®The  Standish  Group,  Charting  the  Seas  oflnformation  Technology  (Dennis,  Mass.:  The 
Standish  Group,  1994). 

^®Capers  Jones,  Assessment  and  Control  of  Software  Risks  (Englewood  Cliffs,  N.J.: 
Yourdon  Press,  1994). 

^^Dean  Leffingwell,  Calculating  the  Return  on  Investment  from  More  Effective 
Requirements  Management  (American  Programmer,  1997)- 
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One  requirement  stated  that  the  "[s]ystem  must  accept  e-filing  documents 
and  case  management  data  in  XML^^  standard  format  and  then  update  the 
case  management  system  with  the  case  filing  data,  case  filing  fees,  and 
store  the  document.”  Deficiencies  in  this  requirement  include  (1)  the  XML 
“standard”  format  is  not  defined,  (2)  the  business  rules  that  should  be  used 
to  compute  the  filing  fees  are  not  referenced,  and  (3)  the  response  time 
and  capacity  (for  example,  number  and  size  of  documents)  was  not 
specified. 

Another  requirement  stated  that  the  system  must  have  the  “[a]bility  to  link 
[a]  case  with  other  related  cases  having  the  same  case  participants  or 
family  unit,  because  the  overall  concept  in  the  family  court  is  one  family, 
one  judge.”  However,  the  document  did  not  specify  such  items  as  (1)  the 
individuals  that  should  be  considered  part  of  the  family  umt  (for  example, 
mother,  father,  stepmother,  aimt,  sister,  nephew,  guardian,  etc.)  or 

(2)  what  are  considered  related  cases  (for  example,  criminal,  civil, 
probate,  etc.). 

A  user  requirement  stated  that  the  system  must  have  the  “[a]bility  to  move 
quickly  between  screens.  Must  have  the  ability  to  change  screen 
navigation  per  each  individual  user  requirement.”  However,  the  term 
“quickly”  was  not  defined  nor  was  there  a  description  of  how  users  should 
define  the  screen  navigation. 

Another  requirement  stated  that  the  system  must  have  the  “[a]bility  to 
electronically  notify  users  based  on  user  defined  rules  when  [a]  user 
defined  event  has  or  has  not  occurred  in  a  timely  manner  by  either  FAX  or 
e-mail.”  Ambiguities  in  this  requirement  include  (1)  the  rules  that  users  can 
define,  (2)  the  functional  method  that  a  user  applies  to  define  a  rule, 

(3)  whether  the  rules  are  established  by  each  user  or  whether  “corporate” 
rules  will  be  defined,  (4)  what  is  considered  “timely,”  and  (5)  who  are 
considered  users  (that  is,  the  reason  why  DC  Courts  personnel  receive 
faxes  instead  of  e-mail  from  their  own  system). 

Another  important  aspect  of  requirements  definition  includes  defining  key 
terms.  In  its  RFP,  the  DC  Courts  did  not  clearly  define  the  terms 
“customization”  and  “modification,”  nor  did  it  require  that  the  vendor 
communicate  the  cost,  schedule,  and  performance  impacts  of 
implementing  a  customization  or  modification  associated  with  a  given 
requirement. 


^^The  Extensible  Markup  Language  (XML)  is  a  flexible,  nonproprietary  set  of  rules  for 
tagging  information  so  that  it  can  be  transmitted  using  Internet  protocols  and  processed  by 
disparate  computer  systems. 
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Unclear  Relationship  to 
Industry  Standards 


The  terms  “customization”  and  “modification”  may  be  confusing  since  a 
basic  premise  is  that  COTS  solutions  should  not  be  customized  or 
modified.  The  terms  “customization”  and  “modification,”  as  used  in  this 
report,  are  differentiated  from  the  basic  installation  processes  that  are 
required  for  systems  such  as  IJIS.  Those  processes  are  commonly  referred 
to  as  installation  and  configuration.  For  purposes  of  this  report,  we  are 
defining  customization  and  modification  as  follows: 

•  Customization:  The  process  of  setting  parameters  within  the  application  to 
make  it  operate  in  accordance  with  the  entity’s  business  rules. 
Customizations  are  normally  supported  by  the  vendor  in  subsequent 
upgrades  as  part  of  the  normal  upgrade  process. 

•  Modification:  The  process  of  writing  or  changing  code.  Modifications  are 
not  generally  supported  by  the  vendor  in  subsequent  upgrades  as  part  of 
the  normal  upgrade  process. 

A  customization  in  one  package  may  be  considered  a  modification  in 
another  package.  For  example,  the  architecture  of  one  product  may  allow 
the  entity  to  develop  a  report  tailored  to  its  needs  in  such  a  manner  that, 
when  the  system  is  upgraded,  the  report  will  still  be  available  without 
additional  development  work.  However,  the  same  report  in  another  system 
may  need  to  be  rewritten  when  the  system  is  upgraded.  Defining  these 
terms  is  important  to  ensure  that  the  benefits  associated  with  any 
customizations  or  modifications  are  cost-effective  and  that  alternatives  are 
not  available. 


In  a  number  of  cases  the  requirements  in  the  DC  Courts’  draft  RFP  did  not 
directly  relate  to  industry  standards  published  by  the  consortium.  The 
consortium  has  been  tasked  with  developing  guidelines  that  will  help  state 
courts  more  effectively  use  their  financial  and  staffing  resources  to  obtain 
state-of-the-art  computer  systems — either  through  in-house  development 
or  through  procurement  from  software  developers.  In  doing  so,  the 
consortium  has  focused  on  ways  in  which  the  state  courts  can 

•  reduce  the  time  needed  to  obtain  a  new  computer  system, 

•  improve  work  processes,  and 

•  reduce  staffing  requirements. 

Recognizing  the  state  courts’  need  for  functional  standards  in  computer 
systems  and  the  staffing  limitations  that  exist  in  most  state  courts,  the 
consortium  has  developed  a  set  of  functional  standards  for  developing 
new  computer  systems.  Courts  nationwide  would  use  these  standards  to 
define  functional  requirements  for  in-house  systems  development  and 
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RFPs  for  vendor-supplied  systems.  Each  court  must  customize  the 
standards  and  add  terminology,  details,  and  specificity  based  on  local  and 
state  procedures,  policies,  and  customs. 

DC  Courts  officials  and  the  contractor  told  us  that  the  requirements  in  the 
DC  Courts’  draft  RFP  were  based  on  the  published  standards  developed  by 
or  currently  under  development  by  the  consortium.  However,  for  many  of 
the  requirements  in  the  draft  RFP,  there  was  no  direct  link,  or  cross- 
referencing,  to  the  standards.  For  example,  one  standard  calls  for 
generating  a  receipt  and  assigning  a  case  number  when  a  case  file  is 
received,  but  the  draft  RFP  requirement  does  not  track  back  to  the 
industry  standard  and  only  speaks  vaguely  of  notifying  users  and 
generating  receipts.  Table  2  compares  one  of  the  standards  to  some  of  the 
draft  RFP’s  requirements. 


Table  2:  Comparison  of  Industry  Standard  and  RFP’s  Requirement 


Industry  standard _ 

Generate  receipt  for  or  notify  appropriate 
parties  that  case  filing  was  received  and 
accepted,  and  give  them  assigned  case 
number  (notice,  including  electronic 
acknowledgment,  would  apply  primarily 
when  a  case  is  transferred  from  another 
jurisdiction  or  filed  electronically). 


Draft  RFP  requirement _ 

Ability  to  electronically  notify  users  based 
on  user-defined  rules  when  user  defined 
event  has  or  has  not  occurred  in  a  timely 
manner  by  either  FAX  or  e-mail. 

Ability  to  automatically  generate  a  receipt 
upon  the  filing  of  a  notice  of  appeal 
stating  the  case  number,  date,  and  time 
of  filing.  System  should  not  allow  a  case 
to  proceed  until  filing  fee  is  collected. 


Ability  to  issue  a  sequentially  numbered 
receipt  for  the  payment  of  funds  on  a 

^sgecificcasa^^_^^^^_^_^^_^__^^^ 


We  were  also  unable  to  identify  in  the  draft  RFP  items  contained  in  the 
industry  standards  that  applied  to  the  DC  Courts.  For  example,  one 
standard  required  the  system  to  “[gjenerate  locally  defined  case  title  or 
style  ( i.e.,  a  short  phrase  that  identifies  case  and  includes  plaintiff  and 
defendant  names)  from  party  names  and  other  information.”  We  were  not 
able  to  readily  identify  a  related  requirement  in  the  draft  RFP. 

Regardless  of  whether  the  draft  RFP  contained  all  the  industry  standards 
that  were  applicable,  the  real  key  is  that  the  draft  RFP  did  not  provide 
adequate  information  so  that  the  prospective  vendors  and  others  could 
readily  map  systems  built  upon  these  standards  to  the  needs  of  the  DC 
Courts.  This  lack  of  vital  information  could  lead  to  adverse  impacts  as 
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vendors  attempt  to  rework  or  develop  functions  that  were  not  clearly 
understood  initially.  This,  in  turn,  could  lead  to  cost  increases,  schedule 
delays,  and  reduced  performance. 


Poor  Organization 
Prevents  Requirements 
Validation 


Requirements  should  be  organized  logically  to  facilitate  imderstanding  and 
requirements  validation  efforts.  The  organization  of  the  requirements  in 
the  draft  RFP  hampers  the  DC  Courts’  and  potential  contractors’  ability  to 
understand  and  validate  the  requirements  since  it  is  very  difficult  to 
determine  where  all  the  requirements  related  to  a  given  functionality  are 
documented.  Because  the  requirements  were  not  logically  organized, 
future  validation  efforts  to  identify  missing  or  duplicate  requirements  may 
not  be  effective. 

Requirements  validation  reduces  requirements-related  defects  by  first 
assessing  whether  the  requirement  document  actually  satisfies  the 
project’s  top-level  requirements.  The  verification  process  includes 
ensuring  that  the 

requirements  document  describes  the  intended  system  behaviors  and 
characteristics; 

requirements  are  correctly  derived  from  requirements  obtained  from  other 
sources,  for  example,  if  the  requirements  document  states  that  a  given 
statute  requires  a  certain  function,  then  an  effort  is  undertaken  to  ensure 
that  the  requirement  correctly  represents  the  specified  statute; 
requirements  are  complete,  consistent,  and  of  high  quality;  and 
requirements  provide  an  adequate  basis  to  proceed  to  the  next  stage  of 
system  development. 


Requirements  validation  is  intended  to  help  ensure  that  the  requirements 
are  complete,  correct,  feasible,  necessary,  prioritized,  unambiguous,  and 
verifiable.^^  The  organization  of  the  requirements  in  the  draft  RFP  will 
likely  hinder  such  a  process.  For  example,  see  the  following: 

•  One  section,  entitled  “Case  Financial  Activity  Requirements,”  contained 
many  of  the  financial-related  requirements  such  as  preparing  a  monthly 
trial  balance  and  balance  sheet.  However,  another  section,  entitled 
“Reporting,”  contained  the  requirement  to  produce  a  Statement  of 
Revenue  and  Expenses.  Generally,  all  requirements  related  to  financial 
statement  reporting  would  be  contained  in  the  same  section. 


^Karl  E.  Wiegers. 


Page  16 


GAO-02-316  DC  Courts 


DC  Coiuts  Has  a 
Reasonable  Basis  to 
Assume  a  Commercial 
Product  Will  Meet  Its 
Needs 


The  requirements  contained  in  the  “Case  Financial  Activity  Requirements” 
section  vi^ere  not  organized  in  a  functional  manner.  Disbursement 
requirements  were  followed  by  adjusting  entry  requirements,  which  were 
followed  by  additional  disbursement  requirements,  which  were  followed 
by  receipt  requirements,  which  were  followed  by  additional  disbursement 
requirements.  In  an  appropriately  organized  requirements  document,  all 
related  requirements  would  be  grouped  together. 

Bar  coding  requirements  for  court  documents  were  contained  in  at  least 
three  different  sections  rather  than  all  in  the  same  section. 


The  DC  Courts  intends  to  use  the  acquisition  process  to  identify  any 
potential  gaps  between  its  system  requirements  and  the  standard  features 
available  in  a  given  vendor’s  product  instead  of  performing  a  detailed  gap 
analysis  before  the  RFP  is  issued.  Although  this  approach  can  increase 
risk,  we  believe  the  DC  Courts’  decision  to  use  the  acquisition  process  to 
identify  the  cost,  schedule,  and  performance  gaps  is  acceptable  in  this 
case  because  of  the  following  mitigating  factors. 

DC  Courts  officials  concluded,  based  on  a  qualitative  analysis,  that  they  do 
not  have  unique  requirements  that  would  prevent  the  utilization  of  a  COTS 
product.  Furthermore,  they  recognized  that  a  gap  analysis  must  be 
performed  as  part  of  the  vendor  selection  process  to  identify  the  cost, 
schedule,  and  performance  impacts  of  each  vendor’s  product  before 
deciding  which  product  to  acquire. 

DC  Courts  is  still  in  the  early  stages  of  the  system  acquisition  effort  and 
has  not  yet  reached  the  critical  point  of  contract  award.  Therefore,  a  gap 
analysis  can  still  be  performed  prior  to  the  system  acquisition  to  ensure 
that  a  COTS  product  can  cost  effectively  meet  DC  Courts’  needs. 

As  noted  previously,  the  requirements  that  had  been  developed  lacked  the 
necessary  specificity  to  perform  a  meaningful  gap  analysis.  Therefore,  the 
time  spent  on  attempting  to  perform  the  gap  analysis  before  issuing  the 
RFP  may  not  have  been  effective. 

As  with  any  effort,  alternative  approaches  need  to  be  analyzed.  In  this 
case,  DC  Courts  officials  believe  that  the  benefits  associated  with  the 
reduced  risks  of  performing  a  detailed  gap  analysis  before  the  RFP  is 
issued — having  a  greater  assurance  that  a  COTS  product  would  meet  their 
needs — ^were  outweighed  by  the  costs  associated  with  delaying  this  effort. 

A  critical  step  in  acquiring  a  new  system  is  determining  whether  the 
system  requirements  can  be  met  by  using  commercially  available  systems, 
commonly  referred  to  as  COTS  systems.  If  COTS  systems  cannot  meet  the 
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majority  of  the  requirements,  then  the  acquirer  will  need  to  undertake  a 
system  development  effort  rather  than  using  a  COTS  product.  To  make 
this  determination,  the  agency  must  perform  a  gap  analysis  that 
systematically  and  quantitatively  compares  and  contrasts  the  vendors’ 
products  against  the  agency’s  requirements  based  on  functional,  technical, 
and  cost  differences. 

Two  different  approaches  can  be  taken  to  perform  this  gap  analysis.  One 
approach  is  to  perform  a  gap  analysis  on  the  detailed  requirements  to 
determine  whether  they  need  to  be  modified  before  issuing  the  RFP  for 
acquiring  a  system.  A  second  approach  is  to  structure  the  acquisition 
process  so  that  the  vendor  identifies  the  gaps  as  well  as  the  cost,  schedule, 
and  performance  impacts  of  addressing  those  gaps.  Each  approach  has  its 
advantages  and  disadvantages.  For  example,  if  a  detailed  gap  analysis  is 
performed  before  the  acquisition  process  begins,  and  it  is  later  determined 
that  a  COTS  product  could  have  cost  effectively  met  the  agency’s  needs, 
then  time  and  money  have  been  wasted  performing  an  analysis  that,  in 
effect,  wUl  be  repeated  during  the  acquisition  process.  On  the  other  hand, 
if  the  second  approach  is  taken  and  the  gap  analysis  discloses  that  a  COTS 
product  cannot  cost  effectively  meet  the  agency’s  needs,  then  the 
acquisition  process  will  need  to  be  restructured  to  support  a  system 
development  effort,  which  translates  into  schedule  delays  and  additional 
costs. 

DC  Courts  officials  selected  the  second  approach — ^using  the  acquisition 
process  to  identify  their  project’s  cost,  schedule,  and  performance  gaps. 
According  to  these  officials,  they  selected  this  approach  because  they  had 

•  reviewed  a  number  of  vendors’  products  and  believed  they  had  a  good 
understanding  of  the  general  capabilities  of  the  marketplace; 

•  hired  a  contractor  that  was  knowledgeable  of  the  court  environment  to 
help  them  understand  whether  a  COTS  product  would  meet  their  needs; 

•  determined  that,  based  on  interactions  with  other  courts  of  similar  size 
and  workload,  the  DC  Courts  can  successfully  use  comparable  practices; 
and 

•  already  committed  to  modifying  their  business  processes  to  reflect  the 
selected  COTS  product’s  capabilities  unless  a  significant  reason,  such  as  a 
legal  requirement,  dictated  otherwise. 

In  discussing  this  issue  with  DC  Courts  officials,  they  acknowledged  that 
the  approach  they  were  taking  increased  risks;  however,  they  believe  that 
the  benefit  obtained — ^accelerating  the  implementation  of  a  badly  needed 
system— justifies  those  risks. 
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Actions  Taken  by  DC 
Courts  to  Address  Our 
Concerns 


In  late  October  2001,  we  discussed  our  findings  on  the  above  three  areas 
with  DC  Courts  officials.  They  generally  agreed  with  our  findings  and 
restated  their  commitment  to  only  go  forward  with  this  project  when  the 
necessary  actions  had  been  taken  to  reduce  the  risks  to  acceptable  levels. 
They  specifically  agreed  to  perform  the  following  actions: 


•  Delay  the  procurement  of  the  new  system  until  the  requirements  can  be 
revised  to  provide  assurance  that  defects  related  to  the  requirements  are 
kept  to  acceptable  levels.  They  have  begun  the  process  of  selecting  a 
contractor  that  will  be  responsible  for  developing  a  new  requirements 
document.  They  also  stated  that  the  contractor  would  be  responsible  for 
documenting  the  requirements  so  that  each  requirement  (1)  fully  describes 
the  system  functionality  to  be  delivered,  (2)  includes  the  source  of  the 
requirement,  and  (3)  is  stated  in  unambiguous  terms  that  allow  for 
quantitative  evaluation. 

•  Develop  a  plan  that  can  be  used  to  guide  DC  Courts’  effort  to  implement 
the  necessary  disciplined  processes.  This  includes  identifying  the  needed 
skUls,  determining  the  best  approach  to  acquiring  the  needed  skills 
through  training  of  DC  Courts’  staff  or  through  contractors,  and  securing 
adequate  funding. 


Conclusions 


DC  Courts’  planned  actions  and  those  taken  thus  far  to  address  our 
findings  are  a  positive  step  forward  and,  if  effectively  implemented,  wiQ 
help  reduce  the  risks  associated  with  this  effort  to  acceptable  levels.  We 
are  reaffirming  these  actions  in  our  recommendations.  Although  the  DC 
Courts  has  a  reasonable  basis  for  believing  that  a  COTS  product  will  meet 
its  needs,  it  has  not  yet  defined  the  requirements  adequately  or 
implemented  the  disciplined  processes  necessary  to  reduce  the  risks  to  the 
project  to  acceptable  levels — ^that  is,  to  reduce  the  risks  associated  with 
disciplined  processes  and  prevent  significant  requirements-related  defects 
in  order  to  limit  the  negative  impact  on  the  cost,  timeliness,  and 
performance  of  the  project.  DC  Courts  has  stated  its  commitment  to 
correctmg  the  deficiencies  in  its  requirements  as  well  as  performing  a  gap 
analysis  during  the  preliminary  phases  of  its  acquisition  project  before  it 
commits  large  amounts  of  resources.  If  properly  implemented,  these  steps 
should  serve  to  reduce  overall  project  risk  and  reduce  the  likelihood  that 
extensive  and  costly  corrections  wiU  be  needed  later. 


Recommendations 


To  help  ensure  that  the  DC  Courts  reduces  risks  associated  with  its 
systems  development  and  implementation  and  increase  the  chances  of  a 
successful  effort,  we  recommend  that  the  Joint  Committee  on  Judicial 
Administration  in  the  District  of  Columbia  take  the  following  actions: 
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DC  Courts  Comments 


Develop  a  plan  on  how  it  will  implement  the  disciplined  processes 
necessary  to  reduce  the  risks  associated  with  this  effort  to  acceptable 
levels.  This  plan  should  include  the  processes,  such  as  those  identified  by 
SEI  and  IEEE,  that  will  be  implemented  and  the  resources,  such  as  staffing 
and  fimding,  needed  to  implement  the  necessary  processes  effectively. 
Develop  requirements  that  contain  the  necessary  specificity  to  reduce 
requirements-related  defects  to  acceptable  levels  and  add  them  to  the  draft 
RFP.  The  requirements  management  process  used  to  develop  and 
document  the  requirements  should  be  adequate  to  ensure  that  each 
requirement  (1)  fully  describes  the  functionality  to  be  delivered, 

(2)  includes  the  source  of  the  requirement,  and  (3)  is  stated  in 
unambiguous  terms  that  allow  for  quantitative  evaluation. 

Organize  the  requirements  document  to  facilitate  the  requirements 
validation  process  used  by  disciplined  organizations. 

Ensure  that  the  acquisition  process  is  adequate  to  (1)  clearly  define  the 
terms  customization  and  modification  and  (2)  ensure  that  vendor 
responses  clearly  communicate  the  cost,  schedule,  and  performance 
impacts  of  implementing  a  customization  or  modification  associated  with 
a  given  requirement. 

Evaluate  the  cost,  schedule,  and  performance  impacts  of  the 
customizations  and  modifications  identified  during  the  system  acquisition 
process  and  ensure  that  the  benefits  are  cost-effective  and  that 
alternatives  to  customization  or  modification  are  not  available. 


In  commenting  on  a  draft  of  our  report,  DC  Courts  agreed  with  our 
findings  and  recommendations  and  said  that  it  has  begun  to  implement  our 
recommendations  to  ensure  the  successful  acquisition  of  UIS  as  outlined 
in  our  report.  Specifically,  DC  Courts  said  that  it  is  implementing  the 
disciplined  processes  critical  to  successful  systems  acquisition.  DC  Courts 
also  noted  that  it  has  a  project  under  way  to  institute  the  necessary 
disciplined  processes  for  the  entire  system  development  life  cycle  (SDLC). 
DC  Courts  further  noted  that  it  is  contracting  with  subject  matter  experts 
in  both  the  SDLC  disciplines  and  court  operations  to  increase  the 
specificity  in  the  RFP  and  to  reduce  the  risks  firom  requirement-related 
defects  to  acceptable  levels.  DC  Courts  provided  additional  details  on  the 
actions  taken  to  address  the  findings  and  recommendations  included  in 
our  report.  If  these  actions  are  successfully  implemented,  they  will 
address  our  concerns  and  help  ensure  the  success  of  the  DC  Courts’  UIS 
acquisition.  The  DC  Courts’  comments  are  reprinted  in  appendix  I. 

We  are  sending  copies  of  this  report  to  the  ranking  minority  member  of 
your  subcommittee  and  to  other  interested  congressional  committees.  We 
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are  also  sending  copies  to  the  Joint  Committee  on  Judicial  Administration 
in  the  District  of  Columbia,  the  chief  judge  of  the  Superior  Court  of  the 
District  of  Columbia,  and  the  executive  director  of  the  District  of 
Columbia  Courts.  Copies  of  this  report  will  also  be  made  available  to 
others  upon  request. 

Please  contact  me  at  (202)  512-9406  or  by  e-mail  at  franzey@gao.gov  if  you 
or  your  staff  have  any  questions  concerning  this  report.  Key  contributors 
to  this  report  were  Linda  Elmore,  John  C.  Martin,  Meg  Mills,  and  Norma 
Samuel. 

Sincerely  yours. 


Jeanette  M.  FYanzel 

Acting  Director,  Financial  Management  and  Assurance 
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Appendix  I:  Comments  from  the  District  of 
Columbia  Courts 


Anne  11.  filiflsB 
Cxcrutciif  (Offirer 


^tsirtet  of  Courts 

500  .Snbtana  A*'en«e,?C*3B*,  |lnnm  1500 
J9laBl]ingtntt,  ®.(C,  20001-2131 

202 

2(1?  879-1820  >'nx 


February  8,  2002 


Jeannette  M.  Franzcl 

Director,  Financial  Management  and  Assurance 
United  States  General  Accounting  Office 
Washington,  DC  20548 

Dear  Ms.  Franzel: 

In  accordance  with  your  request  of  January  25, 2002, 1  enclose  for  your  consideration 
comments  on  behalf  of  the  District  of  Columbia  Courts  to  the  draft  report  entitled  District 
of  Columbia  Courts:  Disciplined  Processes  Critical  to  Successful  System  Acquisition. 

The  District  of  Columbia  Courts  have  carefully  reviewed  the  GAO  report  findings  and 
are  implementing  the  recommendations  to  ensure  the  successful  acquisition  of  the 
Integrated  Justice  Information  System  (IJIS).  If  you  have  further  questions  or  concerns, 
please  contact  me  at  (202)  879-1700. 


Sincerely, 


Anne  B.  Wicks 
Executive  Officer 
D.C.  Courts 
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Appendix  I:  Comments  from  the  District  of 
Columbia  Courts 


COMMENTS  OF  THE  D.C.  COURTS 


According  to  the  General  Accounting  Office  (GAO),  an  initial  review  was  conducted  of 

the  DC  Courts’  effort  to  acquire  a  new  information  system  to  determine  whether: 

1)  DC  Courts  implemented  the  disciplined  process  for  this  report  to  reduce  the  risk 
associated  wi^  this  effort  to  acceptable  levels; 

2)  DC  Courts’  requirements  to  acquire  the  system,  as  identified  in  its  draft  RFP,  contain 
the  necessary  specificity  to  reduce  the  risks  from  requirement  defects  to  acceptable 
levels;  and 

3)  DC  Courts  performed  the  necessary  actions  to  determine  that  a  commercial  off-the- 
shelf  system  (COTS)  would  meet  its  needs, 


COURT  SUMMARY  IN  BRIEF 

1)  The  DC  Courts  have  begun  to  implement  the  disciplined  processes  critical  to 
successful  system  acquisition  as  outlined  in  the  draft  GAO  report.  Additionally,  a 
project  is  underway  which  will  institute  the  necessary  disciplined  processes  for  the 
entire  system  development  life  cycle  (SDLC),  especially  for  the  acquisition, 
installation,  testing  and  acceptance  of  a  new  major  system, 

2)  The  DC  Courts  are  contracting  subject  matter  experts  in  both  the  SDLC  disciplines 
and  court  operations,  to  increase  the  specificity  in  the  RFP  and  to  reduce  the  risks 
from  requirement  defects  to  acceptable  levels.  State  of  the  art  technology  software 
for  requirements  management  will  be  utilized. 


FINDING:  DC  Courts  have  not  implemented  the  disciplined  processes  necessary  to 
reduce  risks  to  acceptable  levels. 

COURT  RESPONSE:  As  indicated  in  the  GAO  report,  the  Court  recognizes  the  need 
for  disciplined  processes  and  plans  on  implementing  such  processes  once  funding  for  the 
integrated  justice  information  system  (IJIS)  project  is  available.  In  an  effort  to  reduce  the 
risks  to  acceptable  levels,  two  measures  are  currently  underway: 

•  At  the  time  of  the  GAO  review,  Court  staff  from  the  Information  Technology 
Division,  Administrative  Services  Division,  and  the  Budget  and  Finance  Division 
were  attending  a  project  management  course  offered  through  the  Boston  University 
Management  Certificate  Program.  In  December  2001  ten  staff  received  their  Project 
Management  certificates.  Another  team  of  personnel  from  the  Courts  are  enrolled 
this  semester.  Additionally,  senior  court  managers  will  receive  project  management 
training  conducted  by  the  Institute  for  Court  Management  from  February  27  through 
March  1,  2002,  further  demonstrating  the  Court’s  commitment  to  implementing  the 
disciplined  processes  necessary  to  reduce  risks  to  acceptable  levels. 
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Appendix  1:  Comments  from  the  District  of 
Columbia  Courts 


•  The  Information  Technology  Division  began  an  analysis  of  software  tools  for  use  in 
the  discipline  of  specification  management,  change  management,  risk  management 
and  testing.  The  Court  selected  the  Rational  Suite  Enterprise  Studio  as  the  tool  to 
attain  the  Capability  Maturity  Model  (CMM),  Using  the  Rational  Unified  Process 
software  the  Court  will  begin  to  move  from  Level  1  (Ad  hoc)  to  Level  2  -  Repeatable 
and  Level  3  -  Defined  maturity  levels  of  the  Software  Engineering  Institute’s  (SEI) 
Capability  Maturity  Model  (CMM).  Also,  the  Court  has  solicited  proposals  for 
consulting  assistance  to  install  the  software,  train  staff,  and  develop  the  necessary 
implementation  procedures.  Throughout  the  acquisition  and  life  cycle  of  the  project, 
the  Court  will  use  the  Rational  Unified  Process  software. 


FINDING:  DC  Courts  system  requirements  as  specified  in  RFP  are  inadequate, 

COURT  RESPONSE:  In  accordance  with  this  finding,  the  Court  has  taken  immediate 
steps  to  adequately  define  the  system  requirements  so  that  they  will  contain  the  necessary 
specificity  and  more  closely  relate  to  the  Court’s  business  operations.  The  National 
Center  for  State  Courts,  our  original  contractor,  will  assist  in  the  refinement  of  the 
systems  requirement.  To  mitigate  risks,  the  Court  will  organize  the  system  requirements 
logically  and  relate  the  requirements  to  industry  standards. 


FINDING:  Unclear  Relationship  to  Industry  Standards 

COURT  RESPONSE:  The  Court  will  revise  the  Request  for  Proposals  (RFP)  utilizing 
the  Case  Management  Functional  Standards  produced  by  the  National  Center  for  State 
Courts  (NCSC),  With  the  assistance  of  the  NCSC,  our  original  contractor,  the  Court  will 
revise  the  RFP  to  include  more  specificity,  including  direct  linkages,  or  cross-referencing 
to  industry  standards.  The  tasks  that  will  be  conducted  to  achieve  industry  standards 
follows: 

I.  Obtain  consulting  assistance  to  draft  a  working  document  (initial  functional 
specifications)  for  users  sessions.  This  would  utilize  information  from  the  current 
RFP,  Volumes  1-4  of  the  business  and  technology  analysis  and  the  current  draft 
and  final  Case  Management  (CM)  Functional  Standards  produced  by  NCSC. 

II.  Conduct  user  sessions  to  review  the  working  document  with  the  various  court 
divisions. 

III.  Draft  a  functional  requirements  document,  using  the  requirements  specification 
tools  which  also  assists  in  the  cross  referencing  of  specifications  to  requirements 
and  organizes  the  information  to  facilitate  the  requirements  validation  process. 

rv.  Revise  RFP  using  new  functional  requirements  document. 

V.  Revise  the  acquisition  process  to  ensure  adequacy  of  clearly  defined  terms  such  as 
customization  and  modification  so  that  vendor  responses  clearly  conmiunicate  the 
cost,  schedule,  and  performance  impacts  of  implementing  a  customization  or 
modification  associated  with  a  given  requirement. 

VI.  Conduct  a  quality  review  by  the  NCSC  and  other  consultant  specialists. 
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Columbia  Courts 


Vn.  Review  comments  by  the  reviewers  to  the  Draft  RFP. 

Vni.  Finalize  RFR 

During  the  vendor  response  evaluation  process,  the  Court  will  evaluate  the  cost,  schedule, 
and  performance  impacts  of  any  customizations  or  modifications  identified  during  the 
system  acquisition  process  to  ensure  that  the  benefits  are  cost  effective  and  that 
alternatives  to  customization  or  modification  are  not  available. 


(194082) 
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GAO’s  Mission 

i - -  ^ 

The  General  Accounting  Office,  the  investigative  arm  of  Congress,  exists  to 
support  Congress  in  meeting  its  constitutional  responsibilities  and  to  help 
improve  the  performance  and  accountability  of  the  federal  government  for  the 
American  people.  GAO  examines  the  use  of  public  funds;  evaluates  federal 
programs  and  policies;  and  provides  analyses,  recommendations,  and  other 
assistance  to  help  Congress  make  informed  oversight,  policy,  and  funding 
decisions.  GAO’s  commitment  to  good  government  is  reflected  in  its  core  values 
of  accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  is  through  the 
Internet.  GAO’s  Web  site  (wwv7.gao.gov)  contains  abstracts  and  fiill-text  files  of 
current  reports  and  testimony  and  an  expanding  archive  of  older  products.  The 

Web  site  features  a  search  engine  to  help  you  locate  documents  using  key  words 
and  phrases.  You  can  print  these  documents  in  their  entirety,  including  charts  and 
other  graphics. 

Each  day,  GAO  issues  a  list  of  newly  released  reports,  testimony,  and 
correspondence.  GAO  posts  this  list,  knovm  as  “Today’s  Reports,”  on  its  Web  site 
daily.  The  list  contains  links  to  the  full-text  document  files.  To  have  GAO  e-mail 
this  list  to  you  every  afternoon,  go  to  www.gao.gov  and  select  "Subscribe  to  daily 
e-mail  alert  for  newly  released  products"  under  the  GAO  Reports  heading. 

Order  by  Mail  or  Phone 

The  first  copy  of  each  printed  report  is  free.  Additional  copies  are  $2  each.  A 
check  or  money  order  should  be  made  out  to  the  Superintendent  of  Documents. 
GAO  also  accepts  VISA  and  Mastercard.  Orders  for  100  or  more  copies  mailed  to  a 
single  address  are  discounted  25  percent.  Orders  should  be  sent  to: 

U.S.  General  Accounting  Office 

P.O.  Box  37050 

Washington,  D.C.  20013 

To  order  by  Phone:  Voice:  (202)  512-6000 

TDD:  (202)  512-2537 

Fax:  (202)  512-6061 

Visit  GAO’s  Document 
Distribution  Center 

GAO  Building 

Room  1100,  700  4th  Street,  NW  (comer  of  4th  and  G  Streets,  NW) 

Washington,  D.C.  20013 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

■ 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm, 

E-mail:  fraudnet@gao.gov,  or 

1-800-424-5454  or  (202)  512-7470  (automated  answering  system). 

Public  Affairs 

Jeff  Nelligan,  Managing  Director,  NelliganJ@gao.gov  (202)  512-4800 

U.S.  General  Accounting  Office,  441  G.  Street  NW,  Room  7149, 

Washington,  D.C.  20548 
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